Summary

Fixes #166242
This is a PR to use an higher level AST instead of the raw ParserTree to compute all Monaco features for ES|QL.

Grammar

This PR drops the existing grammar in favour of the ES one, which a tiny tweak in order to support any case for the query.
I think we could fully adopt the ES grammar without tweaks once this issue is resolved in the antlr4ts dependency.

The adoption of the ES grammar helped also to resolve #166173 .

AST generator

The AST is a subset of the ESQL grammar parser tree, which is easier to navigate.
Tests will be provided to the tool in order to test that the tool can translate even the craziest query string.
Note that this is not preventing the query to be submitted to ES: even if a validation case fails here or the AST fails to translate part of the expression ES will always have the last word about error and warnings, which will be promptly notified to the user.

Commands coverage:

• ROW
  • ROW a=1
  • ROW a=1, b=1
• FROM
  • FROM a
  • FROM a, b
  • FROM a [METADATA ...]
  • FROM a, b [METADATA ...]
    • METADATA won't have indentifiers yet
• SHOW
  • INFO
  • FUNCTIONS
• EVAL
  • EVAL a = b + 1
  • EVAL a = fnA( ... ) + fnB( ... )
  • EVAL b + 1
  • EVAL fnA( ... ) + fnB( ... )
• STATS
  • STATS a = fn(b) + fn(c)
  • STATS a = fn(b) + fn(c) BY group
  • STATS a = fn(b) + fn(c) BY groupA, groupB
  • STATS fn(b) + fn(c)
  • STATS fn(b) + fn(c) BY groupA
  • STATS fn(b) + fn(c) BY groupA, groupB
• WHERE
• LIMIT
• KEEP
• PROJECT
• DROP
• RENAME.
  • RENAME a as b
  • RENAME a + 1 / 5asmyField`
• DISSECT
• GROK
• ENRICH
• MV_EXPAND
• SORT

Collect all user defined variables

• Basic support for assigned variables (i.e. eval a = ...)
• Field type shadowing with assignment
• Advanced support for non-assigned cases? (i.e. eval a + 1)

Validation

With this new approach it would be possible to catch most of warnings and errors while typing, preventing users to submit invalid queries to ES.
Note that here the validation function can detect multiple errors and warnings at once, while ES only returns one error at the time - improving the current sub-optimal editing experience provided.

Tasks

• Create definitions
  • Define function definitions
  • Define aggs definitions
  • Define command definitions
  • Define command options definitions
• Create basic validation walker function
  • Support both errors and warnings
  • Validate commands
  • Validate command options
  • Validate functions
  • Validate source/columns (I phase - just formally)
  • Validate policies (I phase - just formally)
  • Validate literals
  • Validate duration/math syntax
  • Validate variables
• Improve validation work with high level checks
  • Validate source/columns (II phase - leverage ES metadata)
  • Validate policies (II phase - leverage ES metadata)
  • Optimize queries
    • phase I: just make queries strictly when required
• Write tests
  • Source commands
    • Options
  • Processing commands
    • Options
  • Sorting commands
    • All options, combinations
  • Enrich commands
    • Options
  • Warnings cases
• Integrate all of it as Monaco diagnostics module
• Solve issue with long queries and single line mode
• Provide better sync between client and server side error management
  • Show server side warnings only on absence of errors
  • Clear up server side errors on code change

Autocomplete

Using the same provider here to provide better contextual suggestions.
First example giving better suggestions (filtered field based on types and functions based on returned type):

• Commands
  • Show all commands on empty string
  • Show all but source commands after first pipe
  • Suggest option at the right moment

Hover

Basic hover information about used functions:

Things not covered in this PR

• Load function definitions from SHOW FUNCTIONS at language bootstrap
• Make validate function an independent module?
  • Provide a "formal" validation option (i.e. mute Unknown <entity> [...] messages)
• Have autocomplete on par with the current state in main

Validation cases not covered yet

• ... | eval var = 1 year => ❌ it should report an error
• ... | eval 1 + [1, 2, 3] => ⚠️ it should report a null warning
  • same applies for -, *, /, %
• any location aware field/variable check
  • ... | stats a = avg(field) | keep b => ❌ b should be reported as unknown
  • ... | keep a | eval b => ❌ b should be reported as unknown
  • ... | rename a as b, d as c, c as b => ❌ c should be reported as unknown
  • ... | enrich policy on fieldFromEnrich => ❌ fieldFromEnrich is not reported as unknown
  • ... | eval a = round(b), b = round(a) => ❌ b should be reported an unknown
• from ... [metadata <list of meta fields> => ❌ meta fields won't be validated as there's currently no available API to get them

Hover

Advanced usage of hover feature.

Signature

TBD